A METHOD AND SYSTEM FOR IN-SERVICE MONITORING AND TRAINING 
FOR A RADIOLOGIC WORKSTATION 

CROSS REFERENCES to related Applications 

This application is related to the following patent applications, filed at the same 
day as this application and assigned to the same assignee, MeVis Technology GmbH & 
Co. KG 

(1 ) A method and computer system for prefetching of images, Serial No. ; filed 
May 29, 2001 ; 

(2) A method and computer system for screening of medical cases, Serial No. ; 
filed May 29, 2001 ; 

which are hereby incorporated by reference to be understood in connection with the 
present invention. 

Field of the invention 

The invention relates a method and apparatus for use in the field of screening of 
medical cases, and more particularly to training, quality control and quality assurance 
for the screening of medical cases. Further, the invention relates to a method and 
system for in-service monitoring and training for a radiologic workstation. 

Background and prior art 

In a radiologic screening procedure, such as screening mammography, true 
abnormalities such as cancers are believed to occur at a typical rate of about three to 
four cases per one thousand patient examinations. Apparently any misdiagnosis of the 
radiologist can have drastic consequences for the patient. However when a large 
number of cases is screened it is unavoidable that the radiologist's attention decreases 
over time. 



1 



U.S. Patent No. 5,917,929 shows a user interface for a computer aided diagnosis 
(CAD) system. The CAD system serves as an electronic reminder or second reader to 
assist radiologists in obtaining higher detection rates or higher sensitivity for 
abnormalities. In addition the CAD system assists radiologists in reducing the 
misdiagnosis rate, or lowering the false negative rate. However, the usage of a CAD 
system and a user-friendly interface does not address the problem of lapse of the 
attention of the radiologist, which decreases due to fatigue or other reasons. 

U.S. Patent No. 5,987,345 discloses a method and system for displaying of 
medical images. The system displays an image along with corresponding ones of the 
same image with computer-aided diagnostic information added. The CAD computer 
output is used as a second opinion prior to the final decisions of the radiologist. Again 
the problem of a decreasing attention of the radiologist is not addressed here. 

Summary of the invention 

It is an object of the present invention to provide a novel method, apparatus and 
system for screening of medical images. 

It is another object of the present invention to provide for improved training, 
quality assurance and quality control for the screening of medical images. 

It is a further object of the present invention to provide a quality report for the 
screening procedure performed by a radiologist, which can be relied upon by regulatory 
authorities, health insurance providers, and / or legal authorities in case of litigations. 

These and other objects of the invention are achieved by a method, apparatus 
and system which infiltrates known proven positive and/or known normal cases, in the 
sequence of cases to be reviewed, in a fixed or random manner. In brief, the infiltration 
of known positive cases serves to monitor and control the radiologist's attention. This 
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mode of operation is also referred to as "in-service monitoring" or "radiologist's 
performance monitoring". 

Another application of the present invention is for the purposes of training. For 
this application, the radiologist does not review actual cases; rather the radiologist 
reviews known cases having a certain user definable absolute number, statistical 
frequency and/or distribution of a variety of medical case categories. 

The very low incidence rate of 3 to 4 per 1000, make screening mammography 
particularly demanding on radiologist's concentration. The infiltration of known positive 
cases is a way to artificially increase the incidence rate of cancers in screening. 

In a preferred embodiment the absolute number or percentage of known cases to 
be infiltrated is specified by the super user. Further, the categories of known cases to be 
infiltrated can also be selected, as well as, the statistical distribution of various 
categories of the known cases to be infiltrated. 

The invention is advantageous in that it provides an immediate feedback to the 
radiologist during the screening procedure so that the radiologist can recognize a 
decreasing level of concentration and/or fatigue. For example when one or more 
misdiagnosis of known cases occurs, the user can be asked by the system to take a 
break. Alternatively, the system can be disabled for the respective radiologist for a 
certain predetermined time period for appropriate recovery of the radiologist. 

All user actions (e.g. keyboard, mouse clicks and system tools used) concerning 
the real cases to be reviewed, as well as, the known cases and the diagnosis and 
feedback provided to the user can be traced. Based on the tracing, a user action report 
can be generated for the purposes of quality monitoring and control. Such a report can 
even be relied upon for the defence against claims for damages due to a misdiagnosis. 
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Brief description of the drawings 

A more complete appreciation of the invention and many of the attendant 
advantages thereof will be readily understood from the following detailed description of 
preferred embodiments when considered in connection with the accompanying 
drawings, wherein: 

Fig. 1 is a diagram of the system according to the invention; 

Fig. 2 is a diagram illustrating a database for the generation of a user action 

report; 

Fig. 3 is a flow chart illustrating the initialisation of the screening procedure; 

Fig. 4 is a flow chart illustrating an embodiment of the screening procedure in 
accordance with the invention; 

Figs. 5 to 7 are illustrations of screens produced by the system according to the 
invention; 

Fig. 8 is an alternative embodiment of the computer system of the invention 
allowing both screening training and lesion training, 

Fig. 9 is an input screen comprising a screening training sheet, 

Fig. 10 is an input screen comprising a lesion training sheet, and 

Fig. 1 1 is a screen for displaying the training results. 
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Description of the preferred embodiments 

Referring now to the drawings, in particular Fig. 1 , a preferred embodiment of the 
invention will be described. 

Fig. 1 shows a diagram of the computer system according to the invention. The 
computer system has a computer 1 with a user interface 2. For each user of the 
computer 1 , a user profile 3 can be stored. A keypad 4 is connected to the user 
interface 2. The keypad 4 has a next-step button 5. If a user presses the next-step 
button 5, this is a direction to the user interface 2 such that the computer 1 is caused to 
go to the next image or case. 

The user interface 2 is coupled to workflow memory 6, which stores a workflow 
for the screening procedure. The workflow memory 6 contains a case stack 7 for 
storage of the case numbers to be reviewed in a particular screening procedure to be 
performed by a particular user. In the example considered here, the case stack 7 
contains the cases with ID numbers ranging from i to n. The case stack 7 has a pointer 
8 for pointing to the current case that is being reviewed by the user of the computer 1 . 
At the time considered here, the pointer 8 points to the case i, which is the first case of 
the case stack 7. 

The workflow memory 6 is coupled to the image selection module 9 that contains 
a program 10 and a pseudo-random number generator 1 1 . Further the image selection 
module 9 contains a retrieval program module 12 for coupling to a database 13 and for 
retrieval of image data from the database 13. The database 13 can be implemented on 
a mass storage of the computer 1 or on an external server computer, such as an 
external image archive, as is the case in the example of Fig. 1 . 

The database 13 contains a table 14 of cases and the corresponding image data 
for each of the cases. In the example considered here, the table 14 contains the cases 
from case 1 to case m, each of the cases having respective images a, b, c, d. 
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For screening mammography, typically four (or eight) images are taken per case: 
a left and a right craniocaudal image and a left and a right mediolateral oblique image 
(and the priors from the previous screening round). Each of the cases contained in the 
table 14 of the database 13 can be accessed by means of the respective case identifier 
(ID) that serves as a key for performing the database query. 

The image selection module 9 further comprises an infiltration program module 
15 that is coupled to a database 16. Again, the database 16 can be stored on a mass 
storage of the computer 1 , such as a CD-ROM, or it can be stored on a separate server 
computer. 

The database 1 6 has a table 17 for storage of known proven positive and/or 
known proven normal cases. Each of the known cases of the table 17 has a unique 
case ID that serves to retrieve a particular case from the table 17. Each of the cases 
has a data record containing the image data for the case as well as data indicating the 
category of the case. Again, in the case of screening mammography, each case can 
have four images a, b, c, d, corresponding to a left and a right craniocaudal and a left 
and a right mediolateral oblique view, or eight images if the priors from the previous 
round are used also. 

The case category can contain four different data fields corresponding to the 
category of the tumour, the breast density, the mammographic findings (lesion type) and 
a subtlety rating. The table below gives an example of a classification scheme for the 
case category. Each sub-category of a case is identified by a specific value as indicated 
in the table below. 
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Case 
Category 


Name of case category 




1 


Category of tumor 


00 = normal (no tumor) 
1 1 = benign tumor 
22 = cancer screen detected 
33 = interval cancer (over- 
looked, false negative) 

44 = interval cancer (true) 


2 


Breast density 


1 - < 5 % 

2 = 5 - 25 % 

3 = 25 - 50 % 

4 = 50 - 75 % 

5 = > 75 % 


3 


mammographic findings 
(lesion type) 


mass 

calcification 

architectural distortion 

asymmetry 

other (specify...) 

(I)left,(r) right, link to annotations 


4 


subtlety rating 


1 . extremely subtle 

2. very subtle 

3. subtle 

4. relatively obvious 

5. obvious 



Table I 



The infiltration program module 15 can query the database 16 in order to identify 
the known cases in the table 1 7 having a required category or corresponding to a 
certain profile of sub-categories in accordance with the above table I. In the latter case 
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the required case category or sub-category serves as a key to identify suitable cases for 
the infiltration. 

The image selection module 9 is coupled to the session preparation module 18. 
The session preparation module 18 allows specifying the cases to be screened in order 
to initialize the case stack 7. Further the session preparation module 18 allows inputting 
an absolute number or percentage of known cases to be inserted into the flow of cases 
being screened by the user. Further the session preparation module 18 enables 
specifying the absolute number or percentage of known cases of different categories 
and / or different category profiles in accordance with the above table I. The session 
preparation module 18 can serve both to initialize a real screening session as well as a 
training session with training cases. 

The image selection module has an output connected to the display system 1 9, 
which serves to display a current image on the monitor 20. After having reviewed the 
current image, which is displayed on the monitor 20, the radiologist can input a 
diagnosis and / or an annotation via the user interface 2. The diagnosis and / or 
annotation is stored in the diagnosis database 21 of the database module 22. 

The database module 22 further contains a user action database 23 for the 
tracing of user actions that are input via the user interface 2. The user action database 
23 also serves for the purposes of generating the user action report in accordance with 
rules 24. 

It is important to note that the computer system of Fig.1 can be used both for 
training purposes and for the screening of real cases. In the case of training the user 
can initialize the operation of the computer system by means of the session preparation 
module. For example the user can select the number of known cases for the training 
session as well as the statistical distribution of known positive cases. 
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For the purposes of in-service radiologist's performance monitoring of a real 
screening session the session preparation by means of the session preparation module 
18 is typically done by a super user and not by the radiologist who actually performs the 
screening operation. Typically the super user can initialize the computer system for a 
variety of different users, which are identified by respective user IDs. 

For a particular user the case stack 7 is initialized to contain the cases to be 
reviewed by that user as well as a certain profile of known cases to be inserted into the 
flow of images. When the user considered here logs on, the corresponding case stack 7 
and the other entries made by means of the session preparation module 18 by the 
super user are retrieved by means of the user profile 3. 

When the user starts the screening procedure the program 10 obtains a pseudo- 
random number from the pseudo-random number generator 1 1 in order to decide 
whether the real case i to which the pointer 8 points is to be displayed or if a known 
case is to be displayed. If a real case of the case stack 7 is to be displayed the 
database 13 is queried in order to retrieve the case and to display the case. 

If a known case is to be inserted into the flow of cases the program 10 performs 
an access operation to the database 16 via the infiltration program module 15 in order 
to retrieve an appropriate known case for display. It is not transparent to the radiologist 
whether a case that is currently displayed on the monitor 20 is a real case or a known 
case. 

In both instances a diagnosis of the radiologist is entered via the user interface 2 
and stored in the diagnosis database 21 . In the case of a known case the diagnosis is 
compared to the ground truth and/or pathology. If a mismatch between the diagnosis 
and the ground truth and/or pathology occurs, this is recognized by the program 10, and 
a corresponding message is displayed on the monitor 20. For example the message 
can be "you missed a cancer" or similar. 
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The fact that a misdiagnosis occurred is stored in the user action database 23 as 
well as all other user actions. After the user has entered his or her diagnosis for the 
current case he or she can go to the next case by pressing the next-step button 5 such 
that the pointer 8 is shifted to the consecutive case but only if the last case has not been 
a known case. Again the program 10 makes a determination whether to display the 
consecutive case as identified by the pointer 8 or to display a known case from the 
database 1 6. 

This procedure continues until all cases of the case stack 7 have been 
processed. If the number of misdiagnosis recognized by the program 10 gets above a 
certain predefined threshold level the program 10 can display a corresponding message 
to the radiologist and / or it can disable the operation of the computer 1 for a certain 
predefined period of time in order to allow an appropriate recovery of the radiologist. At 
the end of the screening or training session a report can be generated based on the 
contents of the user action database 23 in accordance with the rules 24. For example 
the user action report generation can utilize the following in-service monitoring 
indicators of tables II and III: 



objectives 
(data) 

subjectives 
(diagnosis) 


malignant 


non-malignant 


Finding (lesion visible) 


No finding 
(no lesion 
visible) 


malignant 


benign 


Negative 


Positive 


A 


B1 


B2 


Benign 


C1 


D1 


D2 


Negative 


C2 


D3 


D4 



Table II 
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Variable 


Result 


finding 


malignant 


Comment 


A 


true positive 


YES - ok 


YES - ok 


Lesion found and correctly interpreted as 
malignant 


B1 


False positive 


YES - ok 


YES - not 
ok 


Classification error: a benign lesion is interpreted 
as malignant 


B2 


False positive 


YES - not 
ok 


YES - not 
ok 


something seen that is not there: something 
normal is interpreted as malignant 


C1 


False negative 


YES -ok 


NO - not ok 


Classification error: a malignant lesion is 
interpreted as benign 


C2 


False negative 


NO - not ok 


NO - not ok 


overseen: a malignant lesion is overseen 


D1 


True negative 


YES -ok 


NO -ok 


Lesion found and correctly interpreted as benign 


D2 


True negative 


YES -not 
ok 


NO -ok 


something seen that is not there, but classified as 
benign 


D3 


True negative 


NO - not ok 


NO -ok 


overseen: benign lesion not found, but the result 
is correct (benign) 


D4 


True negative 


NO -ok 


NO -ok 


no finding at all, result is correct 



Table III 



Fig. 2 illustrates an example of the structure of the user action database 23. In 
the example of Fig. 2 the user action database 23 is structured as a table having the 
data fields "user ID", "case ID", "diagnosis", "diagnosis correct?", "user actions" and 
"time spend on case". The data field "user ID" contains the user ID of the radiologist 
who has reviewed a case that is identified by its case ID in the corresponding data field 
"case ID". The diagnostic findings of the radiologist, with the user ID, is indicated in the 
data field "diagnosis". Using the in-service monitoring indicators (A,B1 , etc) the truth of 
this diagnosis can be determined and added in the "diagnosis correct?" column. For 
unknown screening cases, this column is empty. 

The field "user actions" contains a trace of all user actions performed with 
respect to the case with the case ID. Examples for such user actions are the sequence 
of images requested by the radiologist for the review of the case, the format of the 
images requested, such as tiling of the monitor and magnification of the images, as well 
as image transformations and computer aided diagnosis functions which the radiologist 



11 



uses for the review of the case. The following data field "time spent on case" contains 
the amount of time the radiologist has spent on the review of the particular case. 

Fig. 3 shows a flow chart for the initialization of the computer system of Fig. 1 . In 
step 30, the case stack is initialized by specifying the cases to be reviewed by the user. 
Typically, this is done by the super user in the case of a real screening procedure. For 
training purposes this can be done by the user himself. 

In step 31 an absolute number or percentage of known cases to be infiltrated in 
the stream of cases is entered. Again this can be done by the super user or the 
radiologist himself, depending on the application. 

Further in step 32 a distribution of the categories of the known cases can be 
entered such as by specifying a percentage value for each of the categories of above 
table 1 . Further in step 33 the mode can be specified. If the random mode is selected 
the known cases are infiltrated randomly into the stream of cases. If the fixed mode is 
selected the known cases are infiltrated based on a fixed predefined pattern. 

In Fig. 4 the screening procedure is illustrated in greater detail. In step 40 the 
screening procedure is started by the radiologist. In order to display a first case for 
review by the radiologist a pseudo-random number is requested in step 41 . 

Based on the pseudo-random number a decision is made in step 42. If the 
pseudo-random number is equal to 0 this means that a real case from a case stack is to 
be displayed to the radiologist. A corresponding case from the case stack is identified in 
step 43 and displayed to the radiologist in step 44. In step 45 the radiologist can enter 
his or her diagnosis. Alternatively the radiologist can also decide not to enter the 
diagnosis. In this instance the actual case is shifted to the bottom of the case stack (cf. 
Fig. 1 - case stack 7) for later review and diagnosis. 
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In step 46 the user action database 23 is updated for the storage of data 
corresponding to the data fields of the table of Fig. 2 with respect to the case that has 
been reviewed by the radiologist. After step 46 the control returns back to step 41 . 

In step 41 again a pseudo-random number is requested and it is again decided in 
step 42 whether the pseudo-random number equals 0. If the pseudo-random number is 
not equal to 0 the control goes to step 47 in order to select a known case. 

The known case selected in step 47 is displayed in step 48 to the radiologist. 
After review of the case displayed in step 48 the radiologist enters the diagnosis in step 
49. In step 50 it is decided whether the diagnosis entered in step 49 is correct or not. 

If the radiologist classified the known e.g. positive case as a normal case (a 
verified case with no malignant abnormality) or as a negative case (a verified case with 
no abnormality, whether benign or malignant) the diagnosis for the known positive case 
is incorrect and the control goes to step 51 in order to output a corresponding message 
to the radiologist. For example a corresponding message can be displayed and / or an 
aural output such as a beep, can be outputted. 

From step 51 the control goes to step 52 corresponding to step 46 in the case of 
a real medical case. If the diagnosis is correct the control goes directly from step 50 to 
step 52. From step 52 the control goes to step 41 , such that the procedure is repeated 
again. This sequence of steps continues until all cases from the case stack have been 
processed in the screening procedure. 

Figs. 5 to 7 are illustrations of screens produced by the computer 1 of Fig. 1 . Fig. 
5 and Fig. 6 each show a screen of the training session preparation phase. In the 
screen of Fig. 5 the total number of cases to be reviewed can be entered in the input 
field 25. In the example considered here the user can chose between ten and one 
hundred cases. The user's choice is 30 cases in the example considered here. The 
chosen number of cases can be highlighted on the screen. 



13 



Further the user can specify the number of suspicious cases to be infiltrated in 
the case stream. The user can input a minimum number of suspicious cases in the input 
field 26 and the user can input a maximum number of suspicious cases in the input field 
27. If only one of the input fields 26 and 27 is used or if the same number is input in 
both fields 26 and 27 the user can chose between the random and the fixed mode by 
clicking on the word "random" or "fixed". By clicking on "start training" the user can start 
the screening training procedure. 

Fig. 6 shows the screen of Fig. 5 when the user has input different numbers in 
the input fields 26 and 27. In this case the only possible mode is the random mode of 
operation so that the user can directly click on "start training". 

Fig. 7 shows an example of a report, which has been generated based on the 
user action database. The report was done for a screening session of ten cases 1 to 10. 
The result table of Fig. 7 shows the number of correct and incorrect diagnosis as well as 
the number of false positive and false negative cases. A false positive case is a case, 
which is classified by the radiologist as a positive case but which does not contain a 
malignant finding; a false negative case is a case which is classified by the radiologist 
as a negative case, but which does contain a malignant finding. 

Further the average time spent for each individual case is indicated. On request 
a more detailed report can be generated based on the user action database. 

By clicking on "show solutions" the system displays the correct diagnosis of the 
known positive cases. By clicking on "new training set" the control goes back to the 
screens of Fig. 5 and Fig. 6. 

Fig. 8 illustrates an alternative embodiment of the computer system and method 
of the invention allowing both screening training and lesion training. In step 80 of the 
flow chart of Fig. 8 an in-service monitoring statistic is provided. The in-service 
monitoring statistic can be provided by means of a user action report, which has been 
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generated in accordance with step 52 of the flow chart of Fig. 4 (cf. user action data 
base 23 of Fig. 1). 

The in-service monitoring statistic is indicative of the radiologist's performance. 
For example it contains the statistic of the success rate of the radiologist's diagnosis of 
the known cases. The statistic can be specific on the type of cases and can thus serve 
to identify case categories in which the radiologist's diagnoses are frequently incorrect. 

The in-service monitoring statistic is a basis for the mammogram preparation 
(step 81). In step 81 a set of mammograms is selected depending on the in-service 
monitoring statistic provided in step 80. For example if the radiologist's performance is 
particularly weak for a certain case category known cases of this category can be 
selected and prepared for display. 

Alternatively individual settings can be inputted in step 82. By means of the 
individual settings a specific case profile can be specified for selection and preparation 
in step 81. In step 83 a screening training is performed based on the mammograms 
prepared in step 81 . The corresponding cases are displayed in steps 84 and 85 with 
and without computer aided diagnosis (CAD) software support, respectively. The time 
required for the radiologist for inputting his or her respective diagnosis is traced in step 
86. As an option the radiologist gets immediate feedback in step 87, in particular in case 
of a wrong diagnosis. 

The overall result of the screening is evaluated in step 88 and the evaluation is 
displayed to the user. 

As an alternative to the steps 83 to 88 for the screening training a lesion training 
can be performed in step 89. In contrast to the screening training the lesion training 
cases only consist of known positive cases with a variety of different lesion categories to 
be correctly diagnosed by the user. Again the cases are displayed with and without 
CAD support in steps 90 and 91 , respectively. 
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The time required for the diagnosis can be traced in step 92 at the user's option. 
Alternatively the inputting of the diagnosis can be performed in step 93 without time 
control. In either case the radiologist has the option to get immediate feedback in step 
94, in particular in case of a misdiagnosis. In step 95 an overall evaluation of the 
performance of the radiologist for the lesion training is generated and displayed. 

Fig. 9 shows an input screen for specifying a case profile to be reviewed for 
training purposes, such as screening training. The screen can also be used in order to 
specify known positive cases to be infiltrated for the purposes of the in-service 
monitoring. 

The screen allows inputting the total number of cases - in this case 42 cases. A 
slide 96 can be moved along a line 97. The position of the slide 96 determines the 
distribution of normal and positive cases. As the slide 96 is in a middle position the 
same number of normal and positive cases is selected. Moving the slide 96 to the left 
proportionally increases the number of normal cases and decreases the number of 
positive cases such that the total number of cases remains the same. Moving the slide 
96 to the right will correspondingly increase the number of positive cases and 
decreases the number of normal cases. 

Clicking on one of the fields 98, 99 or 100 specifies the age distribution of 
patients from which the mammograms are taken - in this case 50 to 59, years, 60 to 70 
years and greater 70 years. 

Fig. 10 is a corresponding input screen for inputting of individual settings (cf. step 
82). In the case of the lesion training only positive cases (either benign or malign) are to 
be reviewed by definition. The screen shown in Fig. 10 contains an entry field 103 for 
specifying if the current, a previous or a pre-previous mammogram is to be selected. 
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In the entry field 104 the age distribution can be specified. The entry field 105 
serves to specify the type of lesion (mass, microcalcifications, architectural distortion 
(DIST.) asymmetric and other). 

The entry field 106 serves to specify cancer features, such as pathology, tumor 
category and tumor size. The field 107 serves to input the Wolfe class. By clicking on 
the search button 1 08 a number of cases is selected from a case database 
corresponding to the search profile entered by means of the fields 103 to 107. The 
search result is displayed in the window 109. 

Fig. 1 1 shows the result of an overall evaluation. The left column of the screen 
shows the sequence of cases reviewed and the corresponding case IDs. The next 
column shows the corresponding result of the diagnosis. If an incorrect diagnosis has 
been entered the result is "not correctly judged". The right column shows the time 
required for the diagnosis of a particular case. 

Although the present invention has been shown and described with respect to 
preferred embodiments, nevertheless, changes and modifications will be evident to 
those skilled in the art from the teachings of the invention. Such changes and 
modifications that embody the spirit, scope and teachings of the invention are deemed 
to fall within the purview of the invention as set forth in the appended claims. 
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list of reference numerals 



computer 01 

user interface 02 

user profile 03 

keypad 04 

next-step button 05 

workflow memory 06 

case stack 07 

printer 08 

image selection module 09 

program 10 

generator 1 1 

retrieval program module 12 

database 1 3 

table 14 

infiltration program module 15 

database 1 6 

table 17 

session preparation module 18 

display system 1 9 

monitor 20 

diagnosis database 21 

database module 22 

user action database 23 

rules 24 

input field 25 

input field 26 

input field 27 

input field 28 
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